New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ERROR running with gunicorn gevent._gevent_cgreenlet.Greenlet.run #2021
Labels
Status: not gevent
Environment or otherwise not a gevent issue. No further work expected.
Type: Question
User support and/or waiting for responses
Comments
|
jamadden
added
Status: not gevent
Environment or otherwise not a gevent issue. No further work expected.
Type: Question
User support and/or waiting for responses
labels
Feb 13, 2024
I have this same issue with
I have yet to find a consistent repro for this but it goes roughly like this. The Flask app needs to perform a handful of requests, and sometimes, it freezes or throws errors, as mentioned earlier. This is an issue with the opentelemetry instrumentation with gevent workers. # gunicorn.config.py
from gevent import monkey
monkey.patch_all()
import multiprocessing
import os
from opentelemetry import metrics, trace # noqa: E402 monkey patching must be done before importing Instrumentor
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import ( # noqa: E402 monkey patching must be done before importing Instrumentor
OTLPMetricExporter,
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import ( # noqa: E402 monkey patching must be done before importing Instrumentor
OTLPSpanExporter,
)
from opentelemetry.sdk.metrics import ( # noqa: E402 monkey patching must be done before importing Instrumentor
MeterProvider,
)
from opentelemetry.sdk.metrics.export import ( # noqa: E402 monkey patching must be done before importing Instrumentor
PeriodicExportingMetricReader,
)
from opentelemetry.sdk.resources import ( # noqa: E402 monkey patching must be done before importing Instrumentor
Resource,
)
from opentelemetry.sdk.trace import ( # noqa: E402 monkey patching must be done before importing Instrumentor
TracerProvider,
)
from opentelemetry.sdk.trace.export import ( # noqa: E402 monkey patching must be done before importing Instrumentor
BatchSpanProcessor,
)
workers = 1
worker_class = "gevent"
worker_connections = int(os.getenv("GUNICORN_WORKER_CONNECTIONS", "1000"))
timeout = int(os.getenv("GUNICORN_WORKER_TIMEOUT", "600"))
keepalive = int(os.getenv("GUNICORN_WORKER_KEEPALIVE", "60"))
loglevel = "info"
accesslog = "-"
errorlog = "-"
def post_fork(server, worker):
server.log.info("Worker spawned (pid: %s)", worker.pid)
resource = Resource.create(attributes={"service.name": os.getenv("OTEL_SERVICE_NAME", "lih")})
trace.set_tracer_provider(TracerProvider(resource=resource))
span_processor = BatchSpanProcessor(OTLPSpanExporter(insecure=True))
trace.get_tracer_provider().add_span_processor(span_processor)
reader = PeriodicExportingMetricReader(OTLPMetricExporter())
metrics.set_meter_provider(
MeterProvider(
resource=resource,
metric_readers=[reader],
)
) # app.py
from flask import Flask, request
from opentelemetry.instrumentation.flask import FlaskInstrumentor
import os
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
@app.route("/", methods=["POST"])
def process_request():
data = request.get_json()
do_something_with_data(data)
if __name__ == "__main__":
app.run(debug=os.getenv("FLASK_DEBUG", "False").lower() == "true") FROM python:3.10
ENV PYTHONBUFFERED 1
COPY . .
RUN pip install -r requirements.txt
EXPOSE 5000
CMD ["gunicorn", "-c", "gunicorn.config.py", "--bind", "0.0.0.0:5000", "app:app"] |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Status: not gevent
Environment or otherwise not a gevent issue. No further work expected.
Type: Question
User support and/or waiting for responses
Description:
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
File "/usr/local/lib/python3.10/threading.py", line 973, in _bootstrap
self._bootstrap_inner()
File "/usr/local/lib/python3.10/threading.py", line 1008, in _bootstrap_inner
del _limbo[self]
KeyError: <Thread(OtelBatchSpanProcessor, stopped daemon 139660299675232)>
2024-02-08T13:16:51Z <Greenlet at 0x7f05328d2660: <bound method Thread._bootstrap of <Thread(OtelBatchSpanProcessor, stopped daemon 139660299675232)>>> failed with KeyError
IMPORTANT: Please DO NOT POST SCREENSHOTS OF TEXT. Copy and paste
the actual text. This means: No screenshots of your editor. No
screenshots of tracebacks or console output.
What I've run:
REPLACE ME: Paste short, self contained, correct example code (See
http://sscce.org), tracebacks, etc, here
"Put python code in python blocks"
The text was updated successfully, but these errors were encountered: