- Changed all code in the R package to retry failed HTTP requests. This should make
uptasticsearchmore resilient to transient network issues.
- #161 Added support for Elasticsearch 7.x. The biggest changes between that major version and 6.x were the removal of
_allas a way to reference all indices, changing the response format ofhits.totalinto an object like{"hits": {"total": 50}}, and restricting all indices to have a single type of document. More details can be found at https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html.
- #136 removed calls to
closeAllConnections()in unit tests because they were superfluous and causing problems on certain operating systems in the CRAN check farm.
- #138 changed our strategy for deduping records from
unique(outDT)tounique(outDT, by = "_id"). This was prompted by Rdatatable/data.table#3332 (changes indata.table1.12.0), but it's actually faster and safer anyway!
- #64 added support for Elasticsearch 6.x. The biggest change between that major version and v5.x is that as of Elasticsearch 6.x all requests issued to the Elasticsearch HTTP API must pass an explicit
Content-Typeheader. Previous versions of Elasticsearch tried to guess theContent-Typewhen none was declared - #66 completed support for Elasticsearch 6.x. Elasticsearch 6.x changed the supported strategy for issuing scrolling requests.
uptasticsearchwill now hit the cluster to try to figure out which version of Elasticsearch it is running, then use the appropriate scrolling strategy.
- previously,
get_fields()broke on some legacy versions of Elasticsearch where no aliases had been created. The response on the_cat/aliasesendpoint has changed from major version to major version. #66 fixed this for all major versions of Elasticsearch from 1.0 to 6.2
- previously, if you had multiple aliases pointing to the same physical index,
get_fields()would only return one of those. As of #73, mappings for the underlying physical index will now be duplicated once per alias in the table returned byget_fields().
- as of #64,
uptasticsearchattempts to query the Elasticsearch host to figure out what major version of Elasticsearch is running there. Implementation errors in that PR led to versions being parsed incorrectly but silently passing tests. This was fixed in #66. NOTE: this only impacted the dev version of the library on Github.
- In previous versions of
uptasticsearch, the value passed toes_search()forignore_scroll_restrictionwas not actually respected. This was possible because an internal function had defaults specified, so we never caught the fact that that value wasn't getting passed through. #66 instituted the practice of not specifying defaults on function arguments in internal functions, so similar bugs won't be able to silently get through testing in the future.
- #69 added a deprecation warning on
get_counts(). This function was outside the core mission of the package and exposed us unnecessarily to changes in the Elasticsearch DSL
- #51 changed the parsing strategy for nested data and made it 9x faster than the previous implementation
- Functions that make HTTP calls will now use retry logic via
httr::RETRYinstead of one-shotPOSTorGETcalls
get_fields()returns a data.table with the names and types of all indexed fields across one or more indices
es_search()now accepts anintermediates_dirparameter, giving users control over the directory used for temporary I/O at query time
- Added logic to short-circuit and return early with an informative message if a query matches 0 documents
es_search()executes an Elasticsearch query and gets a data.table
chomp_aggs()converts a raw aggs JSON to data.tablechomp_hits()converts a raw hits JSON to data.table
unpack_nested_data()deals with nested Elasticsearch data not in a tabular formatparse_date_time()parses date-times from Elasticsearch records
get_counts()examines the distribution of distinct values for a field in Elasticsearch