Extreme Gesture / Sound delays
needs info
Eren Padar
Gestures and sounds can take AGES to trigger... from 30 to 60 seconds or longer. At one time gesture performance was instantaneous. Now, sometimes they work, often they don't. What good is a gesture that happens 60 seconds after we give the command? Am wondering what is the bottleneck that seems to be affecting so much of SL performance.
Log In
Maestro Linden
Eren Padar Thanks for the gestures you sent today. I ran some tests with them, following this formula:
- Clear viewer cache on both test machines
- Login 2 agents to the same location
- Agent playing gesture was running Second Life Release 7.1.15.15596336374
- Observing agent, on 2nd machine, was running Firestorm 7.1.13.78266
- Region was on Second Life Server 2025-04-24.14648572313
- After the scene and avatars have finished loading on both viewers, play each of the 5 gestures in serial
- Note if audio, text chat, or animations are out of sync between the two viewers, or if the event order appears incorrect across both viewers
- Exit both viewers, and return to step (1)
Unfortunately, I'm still not able to reproduce this issue on either test machine after about 40 triggers of these test gestures. The gesture playback is never fully 100% in sync between the 2 machines, but it always seems to be within about 100ms.
That said, perhaps you (or the other voters) could gather more data when you do reproduce this bug. Details in sub-comment.
Maestro Linden
Here's some general background on how gestures work:
- The gesture owner's viewer loads the gesture data early on in the session
- When a gesture is triggered, the viewer sends a sequence of individual sound/animation/chat events as though they were timed and triggered manually by the user (e.g. as though the user had clicked 'play inworld' on an animation in their inventory to play an animation).
Here's how each gesture event type is messaged:
- Sound playback event: The viewer sends a SoundTrigger message to the simulator, indicating which sound asset should play and where. The simulator echoes this SoundTrigger message back to all nearby viewers.
- Animation playback event: The viewer sends an AgentAnimation message to the simulator, which in turn triggers an AvatarAnimation message to nearby viewers (including information about the animation being triggered and also the list of other animations active on your avatar)
- Chat event: The viewer sends a ChatFromViewer message to the simulator. The simulator echoes this chat to nearby viewers with a ChatFromSimulator
- When gestures trigger normally, the message from the simulator comes back immediately after the viewer's message to trigger the event.
With that in mind, I can think of 3 ways that a gesture event could be delayed between the time it is triggered and when it is rendered on your viewer:
- The source viewer triggering the gesture doesn't send the message to the viewer on time, or suffers a packet loss event when sending the message. All observing viewers would see the event delayed, and perhaps the source viewer too (I'm not sure if the viewer waits for the message from the simulator or not).
- The source viewer sends the message to the sim on time, but the corresponding echo/broadcast message about the event is not sent to some/all viewers on time. In this case, it's possible that some observing viewers would get the message sooner than others.
- Both the source message from the viewer and the corresponding echo message from the simulator are timely, but the viewer suffers a delay in fetching the sound/animation asset_id referenced in the event.
Continued in sub-comment...
Maestro Linden
Eren Padar Since I seem unable to reproduce this bug at all, perhaps you can do some more digging on your system with live packet inspection to see which of the 3 options is responsible for delayed gesture playback.
There are a variety of tools available for this, but I personally like to use Hippolyzer ( https://github.com/SaladDais/Hippolyzer ) since it includes an SL message template, a nice message filter interface, and will show messages fairly cleanly with low latency. For example, when the viewer plays an animation, Hippolyzer immediately shows the event as:
OUT AgentAnimation [RELIABLE]
# ID: 1938
[AgentData]
AgentID = [[AGENT_ID]]
SessionID = [[SESSION_ID]]
[AnimationList]
AnimID = ca5b3f14-3194-7a2b-c894-aa699b718d1f
StartAnim = 1
[PhysicalAvatarEventList]
TypeData =| None
#TypeData = b''
By looking at the timing of these messages, I think you'll be able to identify whether the issue is (1), (2), or (3) listed above. Hopefully you're able to reproduce this when triggering the gesture locally, so that only 1 viewer message stream needs to be viewed.
Eren Padar
Maestro Linden Thanks for putting the time into testing this out Maestro. With respect, I don't work for Linden Lab, am not paid to help debug Linden Lab code, and in fact have helped draw hundreds of people to Linden Lab by doing what users do: host events, perform live music, build and sell, founded two large groups, support huge charity events, and I currently manage a sim. Deep debugging / testing isn't part of my "job description". ;D
However, I think neither of us are surprised that you couldn't reproduce the gesture error. Such was actually predicted in prior messages. I also mentioned that being able to reproduce the error is irrelevant to logically tracking down its cause.
Helping to logically track down the issue is what I'll do now... and it somewhat references your above three points.
POINT 1: I can state this is most unlikely a packet loss issue, for two reasons:
1) I'm the one who issued the gesture command. If the server is not receiving the packet in the first place, then we have an immediate area to examine: why didn't the server receive my command? And why, of the dozens / hundreds of things I've typed that day, did it fail to receive specifically a GESTURE, as opposed to chat or other commands?
2) If the server didn't receive the packet / gesture, then logically the gesture would never play. But it does eventually manifest... in 30 to 60 seconds. That delay indicates the packet was sent, the packet was received, but the server hit a bottleneck somewhere in fetching and transmitting the gesture elements.
Of significant note: A gesture can consist of one, two, or more elements. Chat, sound, animation, etc. So if that gesture is delayed... and then suddenly plays correctly IN ALL ELEMENTS AND TIMING, then packet loss would not seem to be the issue. So we can discount packet loss as being the issue.
POINT 2: This isn't a matter of just other users not receiving the gesture command echoed from the server. This problem manifests with EVERYONE PRESENT, including the sender. No gesture to anyone. So point two can be discounted.
POINT 3: This is a possibility, and supports my hypothesis of some kind of bottleneck between the sim server and asset server.
cont...
Eren Padar
I will add a POINT 4: PACKET DELAY. It is possible that the gesture command is being delayed between my viewer and the server, and that of course would prevent the gesture from playing. But that would bring up the logical question: What in the system could cause a significant and severe packet delay between the Viewer and Server... a delay of 30 to 60 or more seconds? (And then the packet goes through and the gesture works?)
So we can rule out packet LOSS. We can rule out communication failure between the server and other users. What remains on the table is severe packet delay, or total asset retrieval delay... both of which equate to SYSTEM BOTTLENECKS. And it's those bottlenecks we would be looking for.
That's how I debug code. We consider what might possibly be the problem, then Linden Lab has the task of examining their code and methodology to try and figure out the origin of the problem
In the above logical considerations I've pretty much ruled out packet or asset LOSS, which leaves packet or asset DELAY (ie, bottlenecks). That's a hefty part of the task right there. What remains to track down and correct, is causation.
Eren Padar
P.S. Maestro: " Hopefully you're able to reproduce this when triggering the gesture locally, so that only 1 viewer message stream needs to be viewed."
The reason I'm unwilling to do this is two-fold. First is what I've already stated: hunting down technical data on SL errors is not my job. I'm a user, not an employee.
Secondly though, as I've mentioned before: this error is erratic. Those are the most difficult to find. So I'm to sit around for hours, days, weeks collecting data for Linden Lab so it can do what it will have to do anyway-- hunt down a bottleneck in gesture manifestation?
That simply isn't reasonable nor from my end possible. I am happy to help LL if I possibly can within sensible parameters. But your request goes quite a bit beyond what should be expected of a customer of Second Life. I'm not trying to be stubborn or unreasonable. I'm simply pointing out the request itself is unreasonable, and unlikely to produce viable results that would get you where you're going any faster.
I've logically deduced the problem is not packet or asset loss, but rather delay... just by thinking about the issue. That type of process is how I've always debugged code. What comes next... is LL thinking about what might cause such bottlenecks... and checking the code.
Linn Darkwatch
I've noticed this happen at odd times, totally unpredictable except it seems to be with gestures I haven't used for a while. I can't speak to crowded or uncrowded sim, because why would I want to trigger a gesture in an empty sandbox. Usually I would be at an event where there would be half a dozen to several dozen other people. My feeling is that it is more likely to happen in a crowded area but I have never tracked it, just chalking it up to lag.
kickingfisk Resident
In my case with multi-line gestures, the text don't come in the right order. Sometimes, they're just delayed but come in the right order. Sometimes, they don't come at all. A couple of times, the text comes but no sound or vice versa. Weird.
Sometimes, I think it's a sim restart issue since some delays happen when I work at a newly restarted sim (within minutes), and local messages from others come in late. Even radar wouldn't show people on time.
Sometimes, I think it's a viewer problem too since I moved to Genesis viewer this past year, and I'm experiencing them more there. Could just be a coincidence though.
Maestro Linden
needs info
Maestro Linden
closed
Closing for lack of response.
Maestro Linden
needs info
Hi Eren Padar, the playback of animations and sound effects from gestures depends on your viewer having the associated asset. There can be a delay if the viewer needs to download the asset. Repeated playbacks of a gesture should be fast, as the assets are already in the viewer's local cache.
Is this delay occurring the first time the gesture plays in a session, or do you also see this for repeated playback? If it's the latter, can you share an example gesture which demonstrates this issue?
Eren Padar
Maestro Linden Greetings Maestro! The delay occurs only on the first play. Repeated playbacks are instant. The issue is: why should it take 30 to 60 seconds to retrieve a gesture from the asset server? That seems to me extremely poor server performance... and negates the validity of the gesture.
Example: Someone says something funny. So I type in /haha. A FULL MINUTE LATER my avatar laughs. By that time the joke is long forgotten and the laugh is terribly out-of-place.
In the earlier days of SL, gestures and sounds were practically instantaneous. If using a gesture depends on my Viewer already having the associated asset... what good are gestures? If one hasn't been used in a while it's what... (I'm guessing here)... un-encoded from a slow asset archive, transferred to the main asset server, transferred to the sim server, then transferred to my computer? Because I can't imagine any other method taking 30 seconds to a full minute to play a single gesture.
Seriously speaking: can you imagine the result if every single page on Amazon took 30 to 60 seconds to load? How tolerant would shoppers be of that over a period of time? (I'm thinking competitors would pick up a whole lot of business).
That's the issue (with respect): Acceptable performance of Second Life... vs unacceptable performance. And again I have to ask, what is the asset bottleneck(s) affecting so much of Second Life, from gestures to texture rezzing to mesh rezzing to avatar rezzing? Whatever that bottleneck is, it's excessive and really, in my experience shouldn't be all that difficult to identify and correct. I recognize that Second Life likely has millions of asset requests per day... but so does Amazon. Netflix. Prime. Wal-Mart.
Netflix streams live video in real time. Yet Second Life can't transmit a gesture? ('Scuse please, me asking the blunt questions. Computer consultant for 35 years. Retired, but old business habits of "cutting to the chase" die hard.) ; )
Maestro Linden
I tried the following to measure the delay in sound gesture playback times. I tested with Second Life Release 7.1.15.15596336374 (64bit) on Second Life Server 2025-04-24.14648572313, using a 27 second stereo audio clip that I obtained from archive.org:
- Upload a sound clip that's easy to hear and is just barely under the 30 seconds limit imposed by SL. A large audio clip should have the largest file size and thus take longer to download.
- Create a new gesture which print some text (which should appear immediately) and plays the audio clip
- Clear the viewer cache, and login with the viewer
- Once the scene has loaded, trigger the gesture
- Using a stopwatch, measure the time until the audio is heard
- Repeat steps (3) - (5) a few times
The playback times I got were all faster than I could react to start and stop my timer - all times were below 50ms.
When you try these steps, what times do you get? Also, which viewer are you using? This could be a viewer-side bug.
Maestro Linden
I should add that in my test, I was in a fairly empty sandbox region.
Eren Padar
Maestro Linden Maestro, for some reason responses to my posts here are not going to my email-- thus your lack of response. I don't check this page daily.
I don't disagree it could be a side-bug, but I seriously doubt it's purely viewer oriented. I'm thinking more like a bottleneck somewhere.
As for what Viewer I'm using, Firestorm BETA. I'm a registered BETA tester.
Regarding what results I get regarding gesture loading, that should be apparent from my original post.
Regarding your tests of these items to see if they can be replicated-- you create a BRAND NEW GESTURE then test to see if it works? Of course it will work. Now try calling up a gesture that you haven't used in two years... from a standard computer that isn't connected in the same office as Linden Lab, on a FULL sim with a couple dozen other avatars around. (ie, common Second Life conditions, not a "fairly empty sandbox region"). Then see what happens.
The rule-of-thumb here: Sure, test empty sim conditions vs NORMAL sim conditions. That's standard test procedure. But don't leave out the normal, everyday sim conditions test. With all respect: Geez. What happened to standard debugging procedure?
We haven't had an upvotes of this because I haven't notified my groups of this feedback. (That gets to be a hassle, and I don't have group notifications for all my groups.)
I assure you this issue is REAL, and under no circumstance deserves to be CLOSED. You know me Maestro (check with Dan too). Before I file a report I have taken weeks, months or years to examine the problem, seen it happen over and over, tried to see a pattern, reported data (where possible), and am usually right on the button (as when I reported texture load issues both here an on OpenSim). On top of that I have 35 years experience as a RL corporate computer consultant, installer, coder and troubleshooter, much of which involved tracing down problems just like this. Not once in that 35 years did I fail to trace down, locate and fix a problem. (No brag, just fact.) Why not? Because I was a professional. I never took the easy way out and never gave up. My job and reputation depended on it.
So if I go to the trouble of reporting an issue here, you can be pretty sure the issue is real and oft-repeated throughout SL.
Please don't ignore this. It's a real issue, witnessed by your users every hour of every day.
Maestro Linden
Eren Padar: We still need specific steps to reproduce this issue. If it's specific to a certain old gesture in your inventory, can you please reference it and make it available?
The oldest gesture I have in my own inventory (acquired in 2008) does not reproduce this issue. When I trigger it, the sound and animation play immediately.
Eren Padar
Maestro Linden Hi Maestro. Thanks for the speedy reply. There is no discernible pattern, no problem with a specific gesture, nothing that I can send you that would consistently demonstrate the issue. This can happen with any gesture, at any time. If there were I would surely be happy to send it, but this manifestation is unpredictable and follows no pattern that I can see.
Question: what type of computer are you using, where's its location (inside or outside of SL offices). Basically, are you testing under similar conditions to the everyday SL user... or "privileged" SL employee conditions? I mean no rudeness or insult in that question; it's a logical logistics and performance question.
If I may suggest, I'll repeat something a user recommended in another forum: Instead of using "cookie cutter solutions" and trying to replicate the problem before you trace it down, work from the other end (which is how I've done debugging for over 3 decades): ask yourself, "What could POSSIBLY cause gesture retrieval to fail in the first place?"... and make a list. Check the code to see if there's a bottleneck somewhere.
Ask yourself: "What is required for a gesture to work? From the moment the user enters the gesture command, what are all the processes that are involved in making a simple SOUND/CHAT gesture work? What could possibly delay the accomplishment of that gesture for 30 to to 120 seconds?"
cont
Eren Padar
Asking those kinds of questions and thinking about the matter logically... examining the code to see if it LOOKS right... is an option to demanding a consistently reproducible error (which really, is nice when it happens but is quite often not possible during debugging phase).
Forgive me for putting it this way: This is the difference between professional and "lazy" debugging. Requiring a reproducible error is an ideal situation, but that is not always possible. When it's not, it's time to do things the more intellectually-demanding way... and actually hunt down an error to see if you can find something visibly wrong with either the logic path or the code (or both).
I speak from 35 years of professional coding and debugging... during which many a client reported a bug and with only a general description (after all, they aren't coders or techs, they're end-users) and I would have to check the code and logic with minimal information to go on. Would I find the error? Every time, without exception. (No exaggeration. That was my job.)
Maestro Linden
Eren Padar: Since you're curious, I'm on a MacBook Pro M1 (2021), and connecting through a US residential broadband ISP, using wifi on the local network. This machine has no special access to the simulators (which are on AWS) or the asset CDN (on Akamai).
Eren Padar
Maestro Linden Okay thanks Maestro. I appreciate the answer. I'm not sure how a MacBook Pro M1 compares to what I and others use on an average or how our ping rates compare, but that's okay. Sounds like a fair enough testing system. When you say "local network" is that Linden Lab network or your own? I've no intent but just curiosity regarding testing parameters, to achieve significant accuracy.
Because in truth, a 30 to 60 second (or more) delay in performance is very unlikely to be caused by Internet access. That sounds very much to me like a coding or asset fetching bottleneck (the later seems more likely). Or it could be a combination of the two. There's just no conceivable reason (with today's modern computers and asset servers) that a gesture should take that long to manifest on demand. And since at one time on SL they DID manifest immediately on demand, I have to believe something has changed significantly in Second Life, some coding or information transfer or something totally unknown to users that has botched gesture manifestation. Because once it worked CONSISTENTLY... and now it doesn't.
If that's not believed (and that's understandable since you can't replicate the issue), conduct a user pole and ask your everyday users if they experience significant delays in gesture completion. Ask them if there are long gaps between entering a gesture and that gesture actually working. That would be one of the first things I would do. I wouldn't ask LL people, because if they experienced it, surely someone would have brought it to attention. Poll the everyday guy. Poll the tiny communities where I usually hang out. In fact... that much I will do for you... because I really want to help you solve this. I don't mean to come across as a complainer here; I want something that's broke, fixed. I'm sure LL wants the same thing. So out of three tiny communities I have notice access to two. I'll ask where I can, and ask people to be sure to comment and upvote here if they experience significant unexplained gesture delays.
Hope that helps. I really do want to work WITH Linden Lab to help solve this, as I've done in the past. Solutions benefit all everyone. : ) So my goal at this time is to help you folks realize this is actually happening and isn't just the experience of one person.
Maestro Linden
Eren Padar: We still need steps to reproduce the issue, even if it's "trigger these 100 gestures and observe that x% take an unreasonable amount of time to play". Without this information, we have nothing to work with to diagnose the problem or confirm that any future code change fixed it.
Eren Padar
Maestro, with due respect, I am a professional coder with 35 years experience in the corporate industry. I've written programs that have sold internationally, and had very long lifetimes (considering the industry). I have debugged more code than I can list. So when someone tells me they have "nothing to work with", I have to wonder where they learned coding. Because it is often the case: customers report something is going wrong, but the coder can't replicate it. Erratic bugs are common, and the hardest to find. But they must still be found.
So what we do in such case is consider the tracks and resources that gestures demand when they are triggered. As a simple example: if a sound follow a specific set of steps to be played, then we examine the code that handles every one of those steps to see if there is an ancient bug somewhere. We see if a recent change has been made that caused once-functional code to stop working consistently. We see if there is a bottleneck somewhere (which seems to be the case) that causes an on-demand gesture to take 30 SECONDS or more to manifest. (Really, there are only so many things that can cause such a bottleneck.) So the coder tracks those things to see if something is getting in the way of performance.
Again with all due respect, as a retired businessman I have little tolerance for people telling me something "can't be done"... especially in the area of computers. Because over the years we have seen amazing things done in the computer field. And I assure you at one time or another, someone said, "We can't do that." Then they got their lunch eaten by the person who figured out how to do it.
cont
Eren Padar
cont...
So please consider: even if I sent you a box full of gestures that seem to cause me trouble from time to time, I have already stated the issue is inconsistent. And because of the way SL handles assets, you wouldn't be working with MY gestures; you would be working with a whole NEW set of gestures. Agreed they would be calling on the same assets, but the gesture itself would be a brand new gesture.
Now, to help you to the best of my ability, two gestures have failed more than others. You can check them in my inventory (I grant permission) or I can send you copies. I will send you an IM in-world and you can test them all you want. But here's the thing: the reason I see those gestures fail more than others, is because I use them all the time, almost daily. So of course they're going to be the ones to fail more often. That doesn't mean those particular gestures have a problem. It just means the regularity of use will tend to slam into the bottleneck more often than gestures that are not often used.
That is why debugging is done by using logic and wits, not by relying on replication of an error. This is EXACTLY how Kitely fixed the texture issue. Because they were on Kitely servers they couldn't get textures to fail. But when I sent them the data proving beyond all doubt that there was a texture bottleneck, they began a bug hunt... and found more than one. They fixed it, handed it to Beq of Firestorm, she implemented, and then Kitely checked back with me (the test source) to see the results. It worked. That is proper debugging procedure.
Forgive me being direct: Linden Lab's ancient habit of "Give us something we can replicate" is lazy, bad debugging method. It's unprofessional. It is trying to take the easy way out, instead of the logic-based industry standard method. This is Computer 101 stuff: Bugs cannot always be replicated, SO FOLLOW THE LOGIC STREAM. How are gestures performed? Then check every step along that logic line. Sure it takes time. But sometimes that's the ONLY way to find a major issue.