Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hits missing in logs #3

Closed
OkkeKlein opened this issue Feb 4, 2020 · 19 comments
Closed

hits missing in logs #3

OkkeKlein opened this issue Feb 4, 2020 · 19 comments
Assignees
@agazzarini
Labels
bug

Comments

@OkkeKlein
Copy link

Since using the composite requesthandler I am missing the hits counter in Solr logs. Is this possibly related?
@agazzarini
Copy link
Member

Hi @OkkeKlein that's strange. However, let me try; can I ask you which Solr version are you running?

@agazzarini agazzarini self-assigned this Feb 4, 2020
@OkkeKlein
Copy link
Author

OkkeKlein commented Feb 4, 2020 •
edited
Loading

Solr 6.6.6. I have other cores with "normal" requesthandlers that do give hits counter back.

@agazzarini
Copy link
Member

Hi @OkkeKlein I tried to reproduce the issue but unfortunately I haven't been able to do that.
This is what I did.

1) Clone the Solr-Mini-Debugger project
This is a Maven project which contains a sample Solr project with a Test. Read here for more information about this useful project.

2) Apply some change in the project pom.xml
I've put the required repository and dependency for retrieving the composite handler

<repositories>
    <repository>
        <id>iqrh-mvn-repo</id>
        <url>https://raw.github.com/SeaseLtd/composite-request-handler/maven-repository</url>
        <snapshots>
            <enabled>true</enabled>
            <updatePolicy>always</updatePolicy>
        </snapshots>
    </repository>
</repositories>

...
<dependency>
	<groupId>io.sease</groupId>
	<artifactId>composite-request-handler</artifactId>
	<version>1.0</version>
</dependency>

3) Add "author" field
The solr mini debugger has a very simple schema with two fields: id and title. The three example request handlers in the CompositeHandler's README use an "author" field, as well

<fieldType name="string" class="solr.StrField" indexed="true" stored="true"/>
    <fieldType name="text" class="solr.TextField" positionIncrementGap="100" indexed="true" stored="true">
        <analyzer>
            <tokenizer class="solr.WhitespaceTokenizerFactory"/>
            <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
            <filter class="solr.SynonymGraphFilterFactory" expand="true" ignoreCase="true" synonyms="synonyms.txt"/>
            <filter class="io.sease.labs.solr.SystemOutTokenFilterFactory"/>
        </analyzer>
    </fieldType>

    <field name="id" type="string"/>
    <field name="title" type="text"/>
    <field name="author" type="text"/>

    <uniqueKey>id</uniqueKey>

**4) Modify the Solr mini debugger solrconfig.xml **
Take the 3 request handlers in the CompositeHandler's README (plus the composite handler itself) and replaces the request handler defined in the Solr mini debugger solrconfig.xml
Specifically, the composite handler needs to be declared as default handler:

<requestHandler name="/search" class="io.sease.crh.CompositeRequestHandler" default="true">
   <str name="chain">/rh1,/rh2,/rh3</str>
   <str name="rules">eq1,gt0,always</str>
</requestHandler>

5) Modify the Tests class
There's a minimal Test class which adds two documents and then execute a query.
First we need to take in account the "author" field:

    @Before
    public void indexSomeDocumentsBeforeEachTest() throws Exception {
        getSolrClient().add(asList(
                newDoc("1", "Object Oriented Software Construction", "Bertrand Meyer"),
                newDoc("2", "Design Patterns: Elements of Reusable Object-Oriented Software", "Eric Gamma")));
        getSolrClient().commit();
    } 
    private SolrInputDocument newDoc(final String id, String value, String author) {
        SolrInputDocument doc = new SolrInputDocument();
        doc.setField("id", id);
        doc.setField("title", value);
        doc.setField("author", author);

        return doc;
    }

and finally we can experiment with some query:

    @Test
    public void test() throws IOException, SolrServerException {
        SolrQuery thisUsesTheSecondHandler = new SolrQuery("Object")
                .setFields("*")
                .setRows(10);

        QueryResponse response1 = getSolrClient().query("collection1", thisUsesTheSecondHandler);
        System.out.println(response1);

        SolrQuery thisUsesTheThirdHandler = new SolrQuery("Gamma")
                .setFields("*")
                .setRows(10);

        QueryResponse response2 = getSolrClient().query("collection1", thisUsesTheSecondHandler);
        System.out.println(response2);
    }

6) Run Tests class
You will see the following output (there are a lot of other stuff, you can ignore that):

{responseHeader={status=0,QTime=34},response={**numFound=1**,start=0,docs=[SolrDocument{id=1, title=Object Oriented Software Construction, author=Bertrand Meyer}]}}

...

{responseHeader={status=0,QTime=2},response={**numFound=1**,start=0,docs=[SolrDocument{id=2, title=Design Patterns: Elements of Reusable Object-Oriented Software, author=Eric Gamma}]}}

as you can see the numFound is correctly in the response.

@OkkeKlein
Copy link
Author

I also have the numFound in the response. But as mentioned I am missing these values as hits in Solr logs :) &facet=true&facet.sort=count} hits=4570 status=0 QTime=6 is now &facet=true&facet.sort=count} status=0 QTime=6 for example.

@agazzarini
Copy link
Member

Ok, got it. Let me do some other try

@OkkeKlein
Copy link
Author

Can you confirm your logging is missing hits also? If not I need to do some more testing on my side.

@agazzarini
Copy link
Member

Yes, I confirm. I don't see the "hits" attribute when the request flows through the composite handler.

@OkkeKlein
Copy link
Author

Is this something you expect to fix soon? If not, I need to use an alternative approach.

@agazzarini
Copy link
Member

Hi @OkkeKlein give me a couple of days

@agazzarini agazzarini added the bug label Feb 13, 2020
@agazzarini
Copy link
Member

Reference branch for this issue: fix/3_hits_missing_in_logs

@agazzarini
Copy link
Member

Hi @OkkeKlein could you please give a try? Note: I didn't merge in master so you should build from the reference branch above.

Take your time and please let me know if that works for you. After your confirmation, I will merge the PR [1] in master.

[1] #5

@OkkeKlein
Copy link
Author

@agazzarini I get the DocSlice exception again:

2020-02-17 13:37:42.450 WARN (qtp1543727556-16) [ ] o.e.j.s.ServletHandler Error for /solr/csm_test/select
java.lang.NoSuchMethodError: org.apache.solr.search.DocSlice.(II[I[FJF)V
at io.sease.crh.CompositeRequestHandler.(CompositeRequestHandler.java:67)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:539)
at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:486)
at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:753)
at org.apache.solr.core.PluginBag.createPlugin(PluginBag.java:133)
at org.apache.solr.core.PluginBag.init(PluginBag.java:269)
at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:130)
at org.apache.solr.core.SolrCore.(SolrCore.java:935)
at org.apache.solr.core.SolrCore.(SolrCore.java:830)
at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:950)
at org.apache.solr.core.CoreContainer.lambda$load$5(CoreContainer.java:565)
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:197)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

@agazzarini
Copy link
Member

agazzarini commented Feb 17, 2020 via email

@agazzarini
Copy link
Member

@OkkeKlein I just checked it and I confirm you the repository uses Solr 7.1.0 while if I remember you told me you're using Solr 6.6.

If that is true before building the artifact you should change the version in the project pom.xml.

@OkkeKlein
Copy link
Author

@agazzarini yeah, I that was next on my list to test. But had to reindex and still waiting for that.

Error while building with changed POM:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project composite-request-handler: Compilation failure
[ERROR] /E:/Workspace/files/Java/composite-request-handler-fix-3_hits_missing_in_logs/src/test/java/io/sease/crh/CompositeRequestHandlerTestCase.java:[125,65] incompatible types: long cannot be converted to java.lang.Integer

@agazzarini
Copy link
Member

Damn, probably I had the skipTests flag set at built time. I stepped out a bit. I'll fix it as soon as possible.
In the meantime, could you please build using

mvn -DskipTests

@OkkeKlein
Copy link
Author

@agazzarini It looks good so far. Thanx!. I will run more tests.

Does this mean that for every Solr installation with a different version I need to change the Solr version in POM? Or is there a more generic solution?

@agazzarini
Copy link
Member

agazzarini commented Feb 17, 2020 •
edited
Loading

Nice question. Short answer is Yes.

Usually the artefacts (one for each version) are deployed on a maven repository (a popular choice is Nexus) and then retrieved on demand.

We have a mechanism like that implemented here in github for another project (RRE). Since it's a real nightmare from a maintenance point of view, we didn't do that for this repository.

I've created a ticket for that (#6), but I am afraid I cannot assign it a high priority at the moment.

@agazzarini
Copy link
Member

Merged in master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@agazzarini agazzarini

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@OkkeKlein @agazzarini