jnioche merged PR #1343:
URL: https://github.com/apache/incubator-stormcrawler/pull/1343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...
mvolikas commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2466231094
Hi there!
@jnioche I think I have made the changes; also pushed some comments and
minor fixes to the readme files.
I ran some more tests with a greater numbe
jnioche commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2456389209
@mvolikas, latest comments
* need to bring in change from #1390
` storm jar target/crawler-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux
injection.flux --l
mvolikas commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2453478494
> > > We need one for the injection @mvolikas
> >
> >
> > Ok, so I guess I will add this back.
>
> Sorry if I wasn't clear - we need a Flux for the inject
mvolikas commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-245341
An update from my side:
- I have now tested in local mode with 1 and 4 shards.
- I have updated the `SolrSpout` code so that the query param for shards is
not ad
jnioche commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2453474014
> > We need one for the injection @mvolikas
>
> Ok, so I guess I will add this back.
Sorry if I wasn't clear - we need a Flux for the injection, not the Java
mvolikas commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2453471066
> We need one for the injection @mvolikas
Ok, so I guess I will add this back.
--
This is an automated message from the Apache Git Service.
To respond to the mess
mvolikas commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1827008942
##
external/solr/archetype/src/main/resources/archetype-resources/src/main/java/CrawlTopology.java:
##
Review Comment:
Thanks, it's clear now. I h
rzo1 commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2453450532
Bascially, the Java topologies are only good for testing in local mode
(IMHO) and are actually only usable, if the IDE is configured to include the
`provided` scoped dependen
jnioche commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2453448149
> The actually issue is, that the template misses an import for `import
org.apache.stormcrawler.ConfigurableTopology;` -> #1389
The fact that it has been broken for
jnioche commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2453448482
> Yes I have not yet tested with the java topologies
The compilation fails whether you use the Java topologies or not...
--
This is an automated message from t
rzo1 commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1826993141
##
external/solr/archetype/src/main/resources/archetype-resources/src/main/java/CrawlTopology.java:
##
Review Comment:
It also requires the missing im
rzo1 commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2453444795
>
> ```
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
/home/markos/apache/test/src/main/java/tes
rzo1 commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1826991828
##
external/solr/archetype/src/main/resources/archetype-resources/src/main/java/CrawlTopology.java:
##
Review Comment:
This class file needs to be ren
mvolikas commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2453435316
> @mvolikas compiling the project generated from the archetype fails with
>
> ```
> [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin
jnioche commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2453145042
@mvolikas The Java based topologies could actually go. We don't have them in
the OpenSearch module and I think the huge majority of people just rely on the
Flux files.
jnioche commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2453064634
@mvolikas compiling the project generated from the archetype fails with
```
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.11.0:
jnioche commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2453063805
archetype generated successfully, no idea why it had failed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
mvolikas commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1826565193
##
external/solr/src/test/java/org/apache/stormcrawler/solr/persistence/SpoutTest.java:
##
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Fo
mvolikas commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2452974989
> Running the archetype generation with
>
> mvn archetype:generate -DarchetypeGroupId=org.apache.stormcrawler
-DarchetypeArtifactId=stormcrawler-solr-archetype
-Da
jnioche commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1818169400
##
external/solr/README.md:
##
@@ -1,117 +1,30 @@
-stormcrawler-solr
-==
+# stormcrawler-solr
-Set of Solr resources for StormCrawler
jnioche commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1818167365
##
external/solr/archetype/src/main/resources/archetype-resources/README.md:
##
@@ -0,0 +1,144 @@
+This has been generated by the StormCrawler Maven Arc
jnioche commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2437747972
> > @mvolikas Do you aim to include this PR in **3.1.1** - if so, do you
think you can work on the open comments or would it be ok to move it to
**3.1.2**?
>
> From
rzo1 commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2434597624
@mvolikas Do you aim to include this PR in **3.1.1** - if so, do you think
you can work on the open comments or would it be ok to move it to **3.1.2**?
--
This is an automa
mvolikas commented on PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#issuecomment-2435491323
> @mvolikas Do you aim to include this PR in **3.1.1** - if so, do you think
you can work on the open comments or would it be ok to move it to **3.1.2**?
From my si
mvolikas commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1806320925
##
external/solr/setup-solr.sh:
##
Review Comment:
In our team, we currently have an ES deployment for StormCrawler. The main
reason for not usin
rzo1 commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1800926410
##
external/solr/setup-solr.sh:
##
Review Comment:
Guess it would be good to have an archetype. We can still remove it, if it
gets dormant. At least
jnioche commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1800472853
##
external/solr/src/test/java/org/apache/stormcrawler/solr/persistence/SpoutTest.java:
##
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Fou
jnioche commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1800469984
##
external/solr/src/main/java/org/apache/stormcrawler/solr/persistence/SolrSpout.java:
##
@@ -156,10 +176,15 @@ else if (resetFetchDateAfterNSecs != -1
jnioche commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1800465804
##
external/solr/src/main/java/org/apache/stormcrawler/solr/persistence/SolrSpout.java:
##
@@ -135,7 +157,8 @@ else if (resetFetchDateAfterNSecs != -1)
jnioche commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1800461361
##
external/solr/setup-solr.sh:
##
Review Comment:
or in a sub directory?
I am not against having an archetype for SOLR but so far there has bee
mvolikas commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1799842927
##
external/solr/src/test/java/org/apache/stormcrawler/solr/persistence/SpoutTest.java:
##
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Fo
mvolikas commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1799838080
##
external/solr/src/main/java/org/apache/stormcrawler/solr/persistence/SolrSpout.java:
##
@@ -156,10 +176,15 @@ else if (resetFetchDateAfterNSecs != -
mvolikas commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1799832189
##
external/solr/src/main/java/org/apache/stormcrawler/solr/persistence/SolrSpout.java:
##
@@ -135,7 +157,8 @@ else if (resetFetchDateAfterNSecs != -1)
mvolikas commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1798311949
##
external/solr/src/main/java/org/apache/stormcrawler/solr/persistence/SolrSpout.java:
##
@@ -135,7 +157,8 @@ else if (resetFetchDateAfterNSecs != -1)
mvolikas commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1798303419
##
external/solr/setup-solr.sh:
##
Review Comment:
Hmm, you are right. But the alternative would be to have the user copy the
scripts and the `so
mvolikas commented on code in PR #1343:
URL:
https://github.com/apache/incubator-stormcrawler/pull/1343#discussion_r1798272208
##
external/solr/setup-solr.sh:
##
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreem
37 matches
Mail list logo
