Which is literally the same as using the built-in Unix echo tool, but slower and less efficient: $> echo 'hello world' For example: $> echo 'hello world' | bin/echo If the latter is empty then data is written to STDOUT. If the former is empty then data is read from STDIN. The echo tool has two parameters: -from and -to which are pointers to data sources (or “blobs”) to read from and write to, respectively. What distinguishes the echo tool is that it uses the Go Cloud blob package which allows it to read and write data from a variety of different sources and storage providers. There are lots of tools, many of them bundled with a computer’s operating system, that do this already. The third tool is called echo and copies (or echoes) one data source to another. Negative: San Francisco Airport, radar tower. As an added bonus, the exports themselves are in an easy-to-read and easy-to-parse format that we can use to inspect and manipulate outside of Elasticsearch itself. In the end they made our Elasticsearch migration faster than the official snapshot-and-restore model. That’s all these tools do but we think they are useful (and save time) all the same. The restore tool is also able to read data from STDIN so if we wanted to we could pipe the output of the dump tool directly in to the restore tool, migrating one Elasticsearch index to another without the need to write any files to disk. Go does have support for decoding bzip2-encoded files so the restore tool can read the file we just exported by setting the is-bzip flag. That’s why we need to pipe the output of the dump tool to the bzip2 utility. | bzip2 -c > /usr/local/data/millsfield.bz2Īs of this writing the Go programming language doesn’t have native support for bzip2 encoding. Both tools are written in the Go programming language which allows them to be compiled and distributed as standalone binary applications with no external dependencies.īy default the dump tool outputs everything to STDOUT which means you can pipe the data in to other tools like bzip2 to compress it before writing everything to a file on disk. The second tool, called restore, reads a file containing line-separated JSON records and inserts them in to an Elasticsearch index. The first tool, called dump, exports all the records in an Elasticsearch index as a stream of line-separated JSON records that can be written to a file. Elasticsearch has its own process for creating and restoring snapshots of an index but it is sufficiently complex and time-consuming that I decided to write my own tools to perform the migration instead. Recently we needed to migrate 11 million records from one Elasticsearch instance to another. Negative: San Francisco International Airport (SFO), runway or taxiway construction. These tools aren’t specific to SFO Museum and might be useful to other organizations in the sector. In that spirit this blog post is about four small command-line utilities we’ve written to help us do our work and are sharing with the wider cultural heritage sector. We need to learn how to scope the purpose of and our expectations of any single tool so that we can be generous of, and learn from, the inevitable missteps and false starts that will occur along the way. It needs them in the short-term to develop the skill and the practice required to make those tools successful. It needs them in the long-term to finally reach the goal of a common infrastructure that can be employed sector-wide. The cultural heritage sector needs as many small, focused tools as it can produce. As a prelude to that wrap up I want to touch on something I said in the Geotagging Photos at SFO Museum, Part 1 – Setting the Stage blog post: I still need to finish the series of posts we started about our work geotagging photos in the SFO Museum collection and this will happen soon. It’s been a little while since I’ve written anything here. Negative: San Francisco International Airport (SFO), landscape maintenance.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |