[FIXED] How to restrict fields not to get stored in the elastic search database using mapping

Issue

I have a mapping for an index (some_index) as mentioned below.

 {
      "_source": {
        "enabled": false
      },
      "properties": {
        "actions": {
          "store": true,
          "type": "integer"
        },
         "identifier": {
          "store": true,
          "type": "keyword"
        },
        "source": {
          "properties": {
            "entityType": {
              "store": true,
              "type": "keyword"
            },
            "topicPrefix": {
              "index": false,
              "type": "keyword"
            }
            "parentJobId": {
              "store": true,
              "type": "keyword"
            }
          }
        }
      }
    }

Sample document

recordJson = {
        "actions": 40442,
        "source": {
            "entityType": "DELIVERY",
            "parentJobId": "a9a65756-4623-4d7b-ac5f-d2077f3509f6",
            "topicPrefix": "dev"
        },
        "identifier": ""
    }

Here I don't want to save the whole source properties into Elastic search DB. But record JSON will contains these fields. I only need to prevent the source properties to get store in the Elastic search db.

Note: Is there a way we can control it using mapping only, with out doing any change in java. I am working on a project where java code is written very generic and we can not do any changes over there. Any input will much appreciated.

Thanks

Solution

Using an ingest pipeline you can have your documents be modified just before getting indexed. In your case, you'd like to remove the source field and its content, so you can easily achieve it with the following pipeline:

PUT _ingest/pipeline/remove-source
{
  "processors": [
    {
      "remove": {
        "field": "source
      }
    }
  ]
}

And then just reference that pipeline when indexing your data

PUT my-index/_doc/1?pipeline=remove-source
{
    "actions": 40442,
    "source": {
        "entityType": "DELIVERY",
        "parentJobId": "a9a65756-4623-4d7b-ac5f-d2077f3509f6",
        "topicPrefix": "dev"
    },
    "identifier": ""
}

If you don't want or can't specify the pipeline at indexing time, you can also configure your index to always run that pipeline when indexing documents. Just run the following command and nothing changes in your indexing code:

PUT my-index/_settings
{
  "index.default_pipeline": "remove-source"
}

Answered By - Val

Answer Checked By - David Goodson (PHPFixing Volunteer)