Generated dojo_tagulous_test_tags slug too long #10819
Description
Be informative
When adding test tags while reimporting and the generated tag slug already exists defectdojo starts to appends a count to the dojo_tagulous_test_tags.slug field (_nnnn) beginning with 1 to 9999. We've reached 10000 now and get value too long errors when reimporting.
Bug description
When truncating the generated slug and append a number larger than four digits the string is 51 characters long. the slug field is character varying(50).
Steps to reproduce
Steps to reproduce the behavior:
- create more than 10k tests in a loop with identical tags
- See error logs of your database or defectdojo-django uwsgi containers
Expected behavior/ possible fix
Check the slug size before importing.
Truncate one more character when reaching this limit or increase the slug fields max length.
We're running defectdojo version 2.37.1.
Logs
uwsgi:
[28/Aug/2024 06:32:29] ERROR [dojo.api_v2.exception_handler:43] value too long for type character varying(50)
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 916, in get_or_create
return self.get(**kwargs), False
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
raise self.model.DoesNotExist(
dojo.models.Tagulous_Test_tags.DoesNotExist: Tagulous_Test_tags matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 97, in execute
raise ex.with_traceback(None)
psycopg.errors.StringDataRightTruncation: value too long for type character varying(50)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 506, in dispatch
response = handler(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/rest_framework/mixins.py", line 19, in create
self.perform_create(serializer)
File "/app/dojo/api_v2/views.py", line 2802, in perform_create
serializer.save(push_to_jira=push_to_jira)
File "/app/dojo/api_v2/serializers.py", line 2605, in save
self.process_scan(auto_create_manager, data, context)
File "/app/dojo/api_v2/serializers.py", line 2568, in process_scan
).process_scan(
^^^^^^^^^^^^^
File "/app/dojo/importers/default_importer.py", line 106, in process_scan
self.parsed_findings = self.parse_findings(scan, parser)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/dojo/importers/default_importer.py", line 311, in parse_findings
self.parsed_findings = self.parse_findings_static_test_type(scan, parser)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/dojo/importers/default_importer.py", line 325, in parse_findings_static_test_type
self.test = self.create_test(self.scan_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/app/dojo/importers/default_importer.py", line 63, in create_test
self.test = Test.objects.create(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/tagulous/models/tagged.py", line 228, in create
getattr(obj, field_name).save()
File "/usr/local/lib/python3.11/site-packages/tagulous/models/managers.py", line 538, in save
new_tags = self._ensure_tags_in_db(self.tags)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/tagulous/models/managers.py", line 569, in _ensure_tags_in_db
db_tag, __ = self.tag_model.objects.get_or_create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 923, in get_or_create
return self.create(**params), True
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 658, in create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python3.11/site-packages/tagulous/models/models.py", line 447, in save
return super(BaseTagModel, self).save(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
self.save_base(
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
updated = self._save_table(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1020, in _save_table
results = self._do_insert(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1061, in _do_insert
return manager._insert(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 97, in execute
raise ex.with_traceback(None)
django.db.utils.DataError: value too long for type character varying(50)
[28/Aug/2024 06:32:29] ERROR [django.request:241] Internal Server Error: /api/v2/reimport-scan/
[pid: 19|app: -|req: -/-] 10.1.2.3 {64 vars in 1393 bytes} [Wed Aug 28 06:32:29 2024] POST /api/v2/reimport-scan/ => generated 59 bytes in 286 msecs (HTTP/1.1 500) 8 headers in 253 bytes (1 switches on core 1)
postgres:
2024-08-28 05:17:33 UTC:10.9.111.73(12487):admin@dd:[10584]:ERROR: value too long for type character varying(50)
2024-08-28 05:17:33 UTC:10.9.111.73(12487):admin@dd:[10584]:STATEMENT: INSERT INTO "dojo_tagulous_test_tags" ("name", "slug", "count", "protected") VALUES ('image_name=dockerregistry.xxxxxxxxxxxxxxxxxxxx.yz_5000/xxxxxxxxxxxx/xxx-service-xxxxxxxxxxxxxx', 'image_namedockerregistryxxxxxxxxxxxxxxxxxxxx__10000', 0, false) RETURNING "dojo_tagulous_test_tags"."id"