oarepo_oaipmh_harvester.transformers.skip_if_owned
oarepo_oaipmh_harvester.transformers.skip_if_owned
skip-if-owned transformer - lets a fallback harvester give way to other harvesters.
Datastream configuration: skip-if-owned{model:"<model>"}.
Put it last in the transformer chain of a harvester that should write a record only when
no other harvester writes the same local record, e.g. a DataCite harvester next to
the harvesters of the repositories that registered the DOIs. Both harvesters must produce
the same local pid for the same record (e.g. doi/<doi>).
The entry is marked as filtered (and so skipped by the oai-service writer, see
OAIServiceWriter) when
another harvester owns the local pid - it has a harvest record of the same model and pid
that is not deleted at its source and whose last write was not skipped. The local pid is
taken from the harvest record of the entry, or from the transformed data via the pid
provider's get_pid_value_from_record, like the writer does.
Every scenario resolves to the owner's data:
- the other harvester comes first - the entry is skipped, also when it is deleted at the source (the owner's record is not deleted);
- this harvester comes first - its record is overwritten by the other harvester, later harvests of the entry are skipped;
- the other harvester's record is deleted at its source - the entry is written again the next time it is harvested (skipped entries are never up to date), restoring the record with this harvester's data.
SkipIfOwnedTransformer
Bases: BaseTransformer
Mark the entry as filtered if another harvester owns its local record.
Source code in oarepo_oaipmh_harvester/transformers/skip_if_owned.py
__init__(model: str, *_args: Any, harvester_id: str, **_kwargs: Any) -> None
Initialize the transformer.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model
|
str
|
Name of the target model ( |
required |
harvester_id
|
str
|
Harvester the entries belong to (set by the harvest task). |
required |
Source code in oarepo_oaipmh_harvester/transformers/skip_if_owned.py
apply(stream_entry: StreamEntry, *_args: Any, **_kwargs: Any) -> StreamEntry
Mark the entry as filtered, with the reason in entry["skip_reason"], if the pid is owned.