| Age | Commit message (Collapse) | Author |
|---|
|
Dereferencing struct ast_autochan.chan without first calling
ast_autochan_channel_lock() is unsafe because the pointer could change at
any time due to a masquerade. Unfortunately, ast_autochan_channel_lock()
itself uses struct ast_autochan.chan unsafely and can result in a deadlock
if the original channel happens to get destroyed after a masquerade in
addition to the pointer getting changed.
The problem is more likely to happen with v11 and earlier because
masquerades are used to optimize out local channels on those versions.
However, it could still happen on newer versions if the channel is
executing a dialplan application when the channel is transferred or
redirected. In this situation a masquerade still must be used.
* Added a lock to struct ast_autochan to safely be able to use
ast_autochan.chan while trying to get the channel lock in
ast_autochan_channel_lock(). The locking order is the channel lock then
the autochan lock. Locking in the other direction requires deadlock
avoidance.
* Fix unsafe ast_autochan.chan usages in app_mixmonitor.c.
* Fix unsafe ast_autochan.chan usages in app_chanspy.c.
* app_chanspy.c: Removed unused autochan parameter from next_channel().
ASTERISK-26867
Change-Id: Id29dd22bc0f369b44e23ca423d2f3657187cc592
|
|
Channel masquerading had a conflict with autochannel locking.
When locking autochannel->channel, the channel is fetched from the
autochannel and then locked. During the fetch, the autochannel -- which
has no locks itself -- can be modified by someone who owns the channel
lock. That means that the value of autochan->channel cannot be trusted
until you hold the lock.
In practice, this caused problems with Local channels getting
masqueraded away while the ChanSpy attempted to get info from that
channel. The old channel which was about to get removed got locked, but
the new (replaced) channel got unlocked (no-op). Because the replaced
channel was now locked (and would never get unlocked), it couldn't get
removed from the channel list in a timely manner, and would now cause
deadlocks when iterating over the channel list.
This change checks the autochannel after locking the channel for changes
to the autochannel. If the channel had been changed, the lock is
reobtained on the new channel.
In theory it seems possible that after this fix, the lock attempt on the
old (wrong) channel can be on an already destroyed lock, maybe causing
a crash. But that hasn't been observed in the wild and is harder induce
than the current deadlock.
Thanks go to Filip Frank for suggesting a fix similar to this and
especially to IRC user hexanol for pointing out why this deadlock was
possible and testing this fix. And to Richard for catching my rookie
while loop mistake ;)
ASTERISK-25321 #close
Change-Id: I293ae0014e531cd0e675c3f02d1d118a98683def
|
|
For the explanation, here is a copy-paste of the review board explanation:
Initially, it was discovered that performing an attended transfer of a
multiparty bridge with a PJSIP channel would cause a deadlock. A PBX thread
started a masquerade and reached the point where it was calling the fixup()
callback on the "original" channel. For chan_pjsip, this involves pushing a
synchronous task to the session's serializer. The problem was that a task ahead
of the fixup task was also attempting to perform a channel masquerade. However,
since masquerades are designed in a way to only allow for one to occur at a
time, the task ahead of the fixup could not continue until the masquerade
already in progress had completed. And of course, the masquerade in progress
could not complete until the task ahead of the fixup task had completed.
Deadlock.
The initial fix was to change the fixup task to be asynchronous. While this
prevented the deadlock from occurring, it had the frightful side effect of
potentially allowing for tasks in the session's serializer to operate on a
zombie channel.
Taking a step back from this particular deadlock, it became clear that the
problem was not really this one particular issue but that masquerades
themselves needed to be addressed. A PJSIP attended transfer operation calls
ast_channel_move(), which attempts to both set up and execute a masquerade. The
problem was that after it had set up the masquerade, the PBX thread had swooped
in and tried to actually perform the masquerade. Looking at changes that had
been made to Asterisk 12, it became clear that there never is any time now that
anyone ever wants to set up a masquerade and allow for the channel thread to
actually perform the masquerade. Everyone always is calling ast_channel_move(),
performs the masquerade itself before returning.
In this patch, I have removed all blocks of code from channel.c that will
attempt to perform a masquerade if ast_channel_masq() returns true. Now, there
is no distinction between setting up a masquerade and performing the
masquerade. It is one operation. The only remaining checks for
ast_channel_masq() and ast_channel_masqr() are in ast_hangup() since we do not
want to interrupt a masquerade by hanging up the channel. Instead, now
ast_hangup() will wait for a masquerade to complete before moving forward with
its operation.
The ast_channel_move() function has been modified to basically in-line the
logic that used to be in ast_channel_masquerade(). ast_channel_masquerade() has
been killed off for real. ast_channel_move() now has a lock associated with it
that is used to prevent any simultaneous moves from occurring at once. This
means there is no need to make sure that ast_channel_masq() or
ast_channel_masqr() are already set on a channel when ast_channel_move() is
called. It also means the channel container lock is not pulling double duty by
both keeping the container locked and preventing multiple masquerades from
occurring simultaneously.
The ast_do_masquerade() function has been renamed to do_channel_masquerade()
and is now internal to channel.c. The function now takes explicit arguments of
which channels are involved in the masquerade instead of a single channel.
While it probably is possible to do some further refactoring of this method, I
feel that I would be treading dangerously. Instead, all I did was change some
comments that no longer are true after this changeset.
The other more minor change introduced in this patch is to res_pjsip.c to make
ast_sip_push_task_synchronous() run the task in-place if we are already a SIP
servant thread. This is related to this patch because even when we isolate the
channel masquerade to only running in the SIP servant thread, we would still
deadlock when the fixup() callback is reached since we would essentially be
waiting forever for ourselves to finish before actually running the fixup. This
makes it so the fixup is run without having to push a task into a serializer at
all.
(closes issue ASTERISK-22936)
Reported by Jonathan Rose
Review: https://reviewboard.asterisk.org/r/3069
........
Merged revisions 404356 from http://svn.asterisk.org/svn/asterisk/branches/12
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@404368 65c4cc65-6c06-0410-ace0-fbb531ad65f3
|
|
There is a lot that could be said about this, but the patch is a big
improvement for performance, stability, code maintainability,
and ease of future code development.
The channel list is no longer an unsorted linked list. The main container
for channels is an astobj2 hash table. All of the code related to searching
for channels or iterating active channels has been rewritten. Let n be
the number of active channels. Iterating the channel list has gone from
O(n^2) to O(n). Searching for a channel by name went from O(n) to O(1).
Searching for a channel by extension is still O(n), but uses a new method
for doing so, which is more efficient.
The ast_channel object is now a reference counted object. The benefits
here are plentiful. Some benefits directly related to issues in the
previous code include:
1) When threads other than the channel thread owning a channel wanted
access to a channel, it had to hold the lock on it to ensure that it didn't
go away. This is no longer a requirement. Holding a reference is
sufficient.
2) There are places that now require less dealing with channel locks.
3) There are places where channel locks are held for much shorter periods
of time.
4) There are places where dealing with more than one channel at a time becomes
_MUCH_ easier. ChanSpy is a great example of this. Writing code in the
future that deals with multiple channels will be much easier.
Some additional information regarding channel locking and reference count
handling can be found in channel.h, where a new section has been added that
discusses some of the rules associated with it.
Mark Michelson also assisted with the development of this patch. He did the
conversion of ChanSpy and introduced a new API, ast_autochan, which makes it
much easier to deal with holding on to a channel pointer for an extended period
of time and having it get automatically updated if the channel gets masqueraded.
Mark was also a huge help in the code review process.
Thanks to David Vossel for his assistance with this branch, as well. David
did the conversion of the DAHDIScan application by making it become a wrapper
for ChanSpy internally.
The changes come from the svn/asterisk/team/russell/ast_channel_ao2 branch.
Review: http://reviewboard.digium.com/r/203/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@190423 65c4cc65-6c06-0410-ace0-fbb531ad65f3
|
