There was a race condition due to the combination of Python's
object ids being re-usable and Jedi persisting such ids beyond
the real lifeteime of some objects. This could lead to the
subprocess' view of the lifetime of `InferenceState` contexts
getting out of step with that in the parent process and
resulting in errors when removing them. It is also possible
that this could result in erroneous results being reported,
however this was not directly observed.
The race was specifically:
- `InferenceState` A created, gets id 1
- `InferenceStateSubprocess` A' created, uses `InferenceState`
A which it stores as a weakref and an id
- `InferenceStateSubprocess` A' is used, the sub-process learns
about an `InferenceState` with id 1
- `InferenceState` A goes away, `InferenceStateSubprocess` A' is
not yet garbage collected
- `InferenceState` B created, gets id 1
- `InferenceStateSubprocess` B' created, uses `InferenceState` B
which it stores as a weakref and an id
- `InferenceStateSubprocess` B' is used, the sub-process re-uses
its entry for an `InferenceState` with id 1
At this point the order of operations between the two
`InferenceStateSubprocess` instances going away is immaterial --
both will trigger a removal of a state with id 1. As long as B'
doesn't try to use the sub-process again after the first removal
has happened then the second removal will fail.
This commit resolves the race condition by coupling the context
in the subprocess to the corresponding manager class instance
in the parent process, rather than to the consumer `InferenceState`.
See inline comments for further details.
I'm not sure where this was used in the past, however it appears
to be unused now. Removing this simplifies a change I'm about to
make to _InferenceStateProcess.
This removes some of the coupling between the management of the
underlying process and the inference state itself, which intends
to enable changing the origin of the id. This will be useful in
the next commit.
This avoids mypy complaining that we need to provide a generic
argument to Popen, which we cannot acctually do as the implementation
of Popen does not inherit from typing.Generic.
* without passing env_vars to create_environment, GeneralizedPopen behavior is same as before fix to issue #1540 (803c3cb271)
* env_vars allows explicit environment variables, per PR #1619 (f9183bbf64)
