[FIXED] How to make logstash listen to a specific port where the python application is running

Issue

I am trying to make the logstash pipeline to listen to port 8003 where i am running my python starlette application.I need to collect the logs and pass it to the elastic db.

My logstash.conf file is

input {
  udp {
    type => "logs"
    port => 8003
  }
 }
output {
  elasticsearch {hosts => ["https://127.0.0.1:9200"]
                index =>"testlogs"
                user => "elastic"
                password => "XXX"
                ssl => false
                ssl_certificate_verification => false
                cacert =>"xxxxx"
                }
  
}

Also the logs from where the my starlette application running is

uvicorn main:app --reload --port 8003
INFO:     Uvicorn running on http://127.0.0.1:8003 (Press CTRL+C to quit)
INFO:     Started reloader process [53528] using WatchFiles
C:\Users\Dell 5300 2in1\AppData\Local\Programs\Python\Python310\lib\site-packages\elasticsearch\_sync\client\__init__.py:395: SecurityWarning: Connecting to 'https://localhost:9200' using TLS with verify_certs=False is insecure
  _transport = transport_class(
INFO:     Started server process [56984]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
2022-10-04 13:30:19,931 - root - INFO - In web socket function
INFO:     ('127.0.0.1', 61257) - "WebSocket /ws/channel/analytics" [accepted]
2022-10-04 13:30:19,931 - uvicorn.error - INFO - ('127.0.0.1', 61257) - "WebSocket /ws/channel/analytics" [accepted]
INFO:     connection open
2022-10-04 13:30:19,936 - uvicorn.error - INFO - connection open

And after running my conf file i got the below logs in logstash terminal

C:\logstash-8.4.2-windows-x86_64\logstash-8.4.2>logstash -f logstash.conf

[2022-10-04T13:29:02,601][INFO ][logstash.runner          ] Log4j configuration path used is: C:\logstash-8.4.2-windows-x86_64\logstash-8.4.2\config\log4j2.properties
[2022-10-04T13:29:02,613][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2022-10-04T13:29:02,617][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.4.2", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-mswin32]"}
[2022-10-04T13:29:02,621][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-10-04T13:29:02,838][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-10-04T13:29:08,033][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2022-10-04T13:29:08,039][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2022-10-04T13:29:08,626][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-10-04T13:29:09,536][INFO ][org.reflections.Reflections] Reflections took 277 ms to scan 1 urls, producing 125 keys and 434 values
[2022-10-04T13:29:09,880][INFO ][logstash.javapipeline    ] Pipeline `.monitoring-logstash` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-10-04T13:29:09,882][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2022-10-04T13:29:09,965][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["https://127.0.0.1:9200"]}
[2022-10-04T13:29:09,965][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://127.0.0.1:9200"]}
[2022-10-04T13:29:09,983][WARN ][logstash.outputs.elasticsearch][main] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure remove `ssl_certificate_verification => false`
[2022-10-04T13:29:10,016][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@127.0.0.1:9200/]}}
[2022-10-04T13:29:10,016][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@127.0.0.1:9200/]}}
[2022-10-04T13:29:10,100][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@127.0.0.1:9200/"}
[2022-10-04T13:29:10,111][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@127.0.0.1:9200/"}
[2022-10-04T13:29:10,114][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (8.3.3) {:es_version=>8}
[2022-10-04T13:29:10,114][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-10-04T13:29:10,123][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.3.3) {:es_version=>8}
[2022-10-04T13:29:10,123][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2022-10-04T13:29:10,184][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-10-04T13:29:10,192][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-10-04T13:29:10,202][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. `data_stream => auto` resolved to `false`
[2022-10-04T13:29:10,210][WARN ][logstash.javapipeline    ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2022-10-04T13:29:10,217][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with `ecs_compatibility => v8`, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2022-10-04T13:29:10,223][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2022-10-04T13:29:10,323][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["C:/sp/logstash-8.4.2-windows-x86_64/logstash-8.4.2/logstash.conf"], :thread=>"#<Thread:0x3e3e04e1@C:/sp/logstash-8.4.2-windows-x86_64/logstash-8.4.2/logstash-core/lib/logstash/java_pipeline.rb:130 run>"}
[2022-10-04T13:29:10,323][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x4c2a7b61 run>"}
[2022-10-04T13:29:11,383][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>1.05}
[2022-10-04T13:29:11,383][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.06}
[2022-10-04T13:29:11,409][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-10-04T13:29:11,466][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2022-10-04T13:29:11,517][INFO ][logstash.inputs.udp      ][main][8556311d3dbc2bb897d0d7e1c280deb2881bca42bf8bf05affdb5f71ef4a6c46] Starting UDP listener {:address=>"0.0.0.0:8003"}
[2022-10-04T13:29:11,636][INFO ][logstash.inputs.udp      ][main][8556311d3dbc2bb897d0d7e1c280deb2881bca42bf8bf05affdb5f71ef4a6c46] UDP listener started {:address=>"0.0.0.0:8003", :receive_buffer_bytes=>"65536", :queue_size=>"2000"}
[2022-10-04T13:29:11,668][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}

But the logs are not written into the elastic db.Am i doing something wrong here?

Solution

According to your logs, Uvicorn running on http://127.0.0.1:8003 it looks like Uvicorn is actually listening for HTTP requests coming on port 8003, no logs are being sent through that port.

The logs are instead written to a file that you can tail with Logstash using the file input:

file {
    type => "logs"
    path => "/path/to/uvicorn.log"
}

Answered By - Val

Answer Checked By - Cary Denson (PHPFixing Admin)