GitHub user dosubot[bot] added a comment to the discussion: Preventing Duplicate Queries During Asynchronous Chart Data Processing
<!-- Answer --> Your Redis-based locking approach is a solid way to prevent duplicate async chart queries in Superset, since the core system does not provide any built-in distributed locking or deduplication for this use case [(source)](https://github.com/apache/superset/blob/829e4d92d91ceae4b43b1ed3b063ffe45377799c/superset/tasks/async_queries.py). Using `SETNX` with an expiration and exponential backoff is a proven pattern. A few things to watch for: if a worker crashes or is killed, the lock may be orphaned until expiration, and if the lock expires before the query finishes, you could get duplicate queries (the "thundering herd" problem). The lock TTL (`LOCK_EXPIRATION_SECONDS`) is critical—too short and you risk duplicate work, too long and you risk lock contention or abandoned locks. Also, since Redis does not guarantee that only the lock owner can release the lock, consider using a unique value (token) per lock and only deleting if the value matches, to avoid accidental unlocks by other workers. For extra robustness, you might look into the Redlock algorithm or implement lock renewal if queries can run longer than your TTL. Monitoring for repeated lock acquisition failures or long waits can help you tune these values and catch issues early. Overall, your solution is a well-motivated extension to Superset’s async query flow and should help reduce DB load under concurrent dashboard access. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/6347754d-8a96-4a17-b2ed-03a2e20e13b9?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/6347754d-8a96-4a17-b2ed-03a2e20e13b9?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/6347754d-8a96-4a17-b2ed-03a2e20e13b9?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/6347754d-8a96-4a17-b2ed-03a2e20e13b9?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/6347754d-8a96-4a17-b2ed-03a2e20e13b9?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/6347754d-8a96-4a17-b2ed-03a2e20e13b9?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/6347754d-8a96-4a17-b2ed-03a2e20e13b9?feedback_type=other)</sup> [](https://app.dosu.dev/a28d3c7e-a9d3-459e-9fb6-3a6f9ff4f357/ask?utm_source=github)& nbsp;[](https://go.dosu.dev/discord-bot) [](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/apache/superset/discussions/34316) GitHub link: https://github.com/apache/superset/discussions/34316#discussioncomment-13901087 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
