If the thread you are trying to stop is in the same process as the proc that is doing the stopping, I don’t think that any timeout will be sufficient. When you do a “system” call, the process will not be able to respond to “hcicmd” commands for the duration of the system call.
You could possibly execute the “hcicmd” in the background (system “hcicmd blah &”), but you would not be able to check the result of the command.